The need for image stabilization arises in many applications including the movie industry which must remove unwanted jitter between successive frames of source video, television news cameramen who must stabilize video from hand held cameras in newscasts produced in the field, video from surveillance cameras molted on swaying or rotating platforms, or on moving vehicles, which must be stabilized prior to computer analysis, or prior to display to human observers, and video from moving vehicles which must be stabilized prior to image compression or presentation to a remote operator (teleoperation) or to computer vision systems for automatic driving.
Mechanically stabilized platforms of various types are used in surveillance to compensate for imager, not image, motion. In FIG. 1, an imager 10 is mounted on a mechanically stabilized platform 12. The output I.sub.in (t) on lead 14 of the imager 10 is displayed on a monitor 16. The platform 12 typically uses gyroscopes to sense platform rotation, and motors to compensate for that rotation. A user can guide the imager direction of gaze (pan and tilt) and zoom via an electronic control sisal 18 and motor drives in the platform 12.
Electronic stabilization with imager motion sensors can be used to compensate for imager motion but not image motion. In FIG. 2, an imager 10, mounted on a mechanically stabilized platform 12, has an output I.sub.in (t) on lead 14. Residual motion of the platform 12 is measured using sensors 20. Sensed displacements d(t) on lead 22 are converted to transformation parameters p(t) on lead 24 by a transform module 26. Parameters p(t) 24 are used by image warp deuce 28 to produce a stabilized output image I.sub.out (t) 30, in which imager motion has been compensated, for display on monitor 16.
In FIG. 3 a system for electronic stabilization with digital processing to sense image motion is shown. The system includes imager 10 having an output I.sub.in (t) 14 which is stored in an image frame store 32 to hold the image until appropriate warp parameters have been computed and transmitted to image warp device 28 for stabilization. (This frame store is not needed if the warp performed at one frame time is based on parameters computed at the previous frame time.) In order to reduce memory, a set of features may be extracted from the source video by module 34. A second image frame store 36 is provided to hold the features extracted from the previous image so that it can be compared with the present image. The features f(t) on lead 38 extracted at time t are compared with features f(t-1) on lead 40 extracted at time (t-1) in displacement estimator 42 that uses digital image processing to determine image-to-image motion to produce displacements d(t) on lead 44. One commercially available camera uses an array of 36 pixels as the features compared between frames. Sensed displacements d(t) on lead 44 are used by image warp device 28 to produce a stabilized output image I.sub.out (t) 30, in which image motion has been compensated, for display on monitor 16. Such systems known to the inventor can not compensate zoom, rotation, parallax and/or lens distortion.
In FIG. 4 an electronic target tracking system uses correlation to locate a target within an imager's field of view, then steers the imager to center the target, and, at least roughly, stabilize the target pattern. The system includes imager 10 having an output image I.sub.in (t) 14 which is compared in a correlation module 46 to match a reference pattern stored therein with the image 14. The reference pattern is selected from a prior frame in the video sequence module 48, stored in memory 50. The reference pattern may represent a stationary object in the scene or a moving object. The difference between the image I.sub.in (t) and the reference pattern provides displacement information to transform module 26 which converts this information to produce a signal to stabilize platform 12.
In FIG. 5 a system for electronic target tracking using change detection is shown. The difference between the current image I.sub.in (t) and a previous image I.sub.n (t-1) in frame store 60 is determined by subtractor 62 and regions of significant change are located. The location information x(t) provided by module 64, is used to redirect the imager 10 to maintain the image at a particular point or on a particular track in the displayed image. This approach may be preferred to a pattern based approach when targets are too small to be detected based on pattern match, but are moving so that detection can be based on this motion. Existing systems require that motion of the background scene be small so that target motion can be detected.
Burt et al. in the Proceedings Of The Workshop On Visual Motion, Irvine, Calif., Mar. 20-22, 1989, pages 1-12, have disclosed procedures for obtaining precise image alignment through iterative refinement. Such procedures have been used for such applications as terrain shape recovery and moving target detection. It would be desirable to have a method and system for electronically stabilizing images which can compensate zoom, rotation, parallax and/or lens distortion while having improved accuracy. It would also be desirable to extend the functionality of an image stabilization system to include derived image sequences with reduced noise or highlighted change by making use of information generated as a byproduct of the improved stabilization process.